Repository navigation
Pruning should affect candidate datasets too - #10877
Conversation
A few weeks ago we were debugging an exhaustive search through all local
storage allocations, and determined it was caused by the invalid
PgNumeric conversion to ByteCount. The next step was to cause that
search to bail out in order to unwind the saga: this was _supposed_ to
be achieved by adjusting `size_used` column of the rendezvous local
storage dataset to a number that would not be affected by the invalid
conversion, and we did this, but observed that the search continued
anyway!
Consider the following incomplete allocation list:
0: allocation0 -> zpool0
1: allocation1 -> zpool1
2: allocation2 -> zpool2
3: allocation3 -> zpool3
4: allocation4 -> zpool4
5: allocation5 -> zpool5
6: allocation6 -> zpool6
7: allocation7 -> zpool7
Note the allocations and zpools here are ordered to make the example
easier but will actually be random depending on the order of rows
returned.
The remaining allocations to perform are 8 and 9, and they have a list
of zpools to try to allocate against. The pathologically bad case is if
any zpool could be a candidate:
allocation8: [zpool0, zpool1, ..., zpool6, ..., zpool8, zpool9]
allocation9: [zpool0, zpool1, ..., zpool6, ..., zpool8, zpool9]
Previous to this commit, the pruning step only affected the list of
incomplete allocations. This doesn't cause the search to bail out! If
zpool6 runs out of space due to concurrent allocations, the pruning step
will not retain incomplete allocations that use zpool6:
0: allocation0 -> zpool0
1: allocation1 -> zpool1
2: allocation2 -> zpool2
3: allocation3 -> zpool3
4: allocation4 -> zpool4
5: allocation5 -> zpool5
6: allocation6 -> zpool6 <- any search that made it this far
7: allocation7 -> zpool7 will be pruned
but the iterator will continue to descend down the search space using
zpool6 because it's still in the candidate list:
0: allocation0 -> zpool0
1: allocation1 -> zpool1
2: allocation2 -> zpool2
3: allocation3 -> zpool3
4: allocation4 -> zpool4
5: allocation5 -> zpool5
allocation6: [zpool0, zpool1, ..., zpool6, ..., zpool8, zpool9]
^- still in the list!
This commit adds a pruning step that removes from an allocation's
candidate list based on the updated ZpoolGetForSledReservationResult
information.
| info!(self.log, "removed {removed} from queue"); | ||
| } | ||
|
|
||
| removed = 0; |
There was a problem hiding this comment.
nit: Can we use a different variable name here? I'd rather these be distinct, rather than trying to rely on "resetting" a single value with two different meanings
| }); | ||
|
|
||
| if removed > 0 { | ||
| info!(self.log, "removed {removed} from queue"); |
There was a problem hiding this comment.
| info!(self.log, "removed {removed} from queue"); | |
| info!(self.log, "removed allocation lists from consideration during instance provisioning"; "count" => removed); |
There was a problem hiding this comment.
changed to be less vague in 0dc86bb, but I didn't separate out the count, I'd rather that be part of the message
| } | ||
|
|
||
| if removed > 0 { | ||
| info!(self.log, "removed {removed} from candidate datasets"); |
There was a problem hiding this comment.
| info!(self.log, "removed {removed} from candidate datasets"); | |
| info!(self.log, "removed candidate datasets from consideration during instance allocation"; "count" => removed); |
| if removed > 0 { | ||
| info!(self.log, "removed {removed} from candidate datasets"); | ||
| } | ||
| } |
There was a problem hiding this comment.
What do you think about re-assessing a more holistic check of "can this set fit" here?
There are a couple checks we perform when initially assembling the CompleteLocalStorageAllocationsList:
"Are there enough pools to satisfy this request"
https://github.com/jmpesp/omicron/blob/5886e0028363594e435a8fdaa1fdc01ba787cfd2/nexus/db-queries/src/db/datastore/sled.rs#L465-L477
"Are there any remaining candidate datasets"
https://github.com/jmpesp/omicron/blob/5886e0028363594e435a8fdaa1fdc01ba787cfd2/nexus/db-queries/src/db/datastore/sled.rs#L505-L516
We could presumably re-check these constraints here, with something like:
let distinct_pools: HashSet<ZpoolUuid> = self
.allocations_to_perform
.iter()
.flat_map(|r| r.candidate_datasets.iter().map(|c| c.pool_id))
.collect();
let infeasible = self
.allocations_to_perform
.iter()
.any(|r| r.candidate_datasets.is_empty())
|| distinct_pools.len() < self.allocations_to_perform.len();
if infeasible {
self.queue.clear();
}Without doing this, I think these cases of "you asked for N disks, but we only have M < N zpools" will be doing a significant amount of work in the invocation of next() within CompleteLocalStorageAllocationLists. Internally, the self.queue.pop() in that next implementation will need to get exhausted before returning, which seems expensive.
For example:
- Suppose we're in that
while let Some(incomplete_allocation_list) = self.queue.pop()loop, and we've madekprojected allocations thus far - For the next request
k, we loop overcandidate_datasets, keeping the ones that are still incandidates_left(which, in this case, would beM - kpools). For each of thoseM-kpools, we push new possibleIncompleteAllocationLists.
Because we have a "loop, pushing children (and doing clones) for each candidate_dataset", I think this is a factorial space exploration before we exhaust the set of candidate allocations.
As an aside: We might want to re-evaluate this whole approach here, and consider a more efficient bipartite matching algorithm to avoid having a worst-case-factorial search here in the first place? I think my proposal would help in cases where we have an instance requesting N homogenously-sized disks, but I'm not sure it works if the local storage disks are differently-sized.
There was a problem hiding this comment.
I think you're right that the iterator logic will spin if someone asked for N disks but we only have M < N zpools, but there's already a Not enough zpools to satisfy the number of allocations required check for that in the iterator's new. I added a test that exercises this in 511cd30
There was a problem hiding this comment.
So the check here covers this issue at construction-time, but my concern is about that infeasible state arising mid-search.
The new pruning logic added in this PR can change the candidate lists into a state that our constructor would have rejected, but it doesn't get a chance to.
Specifically:
- If
candidate_datasetsbecomes empty (which we check in the constructor here), or - The number of distinct pools across all candidate lists is now smaller than the number of requests (which we normally check in the constructor here)
If either of these happen as an outcome of prune_invalidated_allocation_lists, we never re-check them. I'm mostly worried about wasted work in https://github.com/jmpesp/omicron/blob/367f71137c72d64ab13ad8400a0cbe377f24eb27/nexus/db-queries/src/db/datastore/sled.rs#L775
NORMALLY I wouldn't care much; our checks should identify these cases and eliminate them anyway, but the next function in impl<'a> Iterator for CompleteLocalStorageAllocationLists<'a> is an extremely expensive operation: unless the allocation has already succeeded, it's generating quite a bit of additional queue work.
I made a separate comment below about the ..._pruning_{1,2} tests passing the wrong instance IDs - but I tried to check out this branch, fix that, and then compare the cost of "how expensive it it to call CompleteLocalStorageAllocationLists::next() in the "pruning_2" test with / without the check I suggested?
- The PR, as-is (with the correct instances patched):
next()takes 3.34 seconds of pure CPU time. - With the mid-search re-check I suggested:
next()takes 979 nanoseconds, because it's not spending time throwing things into / out of the queue.
There was a problem hiding this comment.
I will also admit: If we think we're going to re-do this whole approach in the near future, we can toss this optimization. We don't need to refine this if we'd prefer to replace the exhaustive search approach instead.
| affinity: None, | ||
| ncpus: 2, | ||
| memory: external::ByteCount::from_gibibytes_u32(16), | ||
| disks: (0..10) |
There was a problem hiding this comment.
I think the pessimistic case I described above could be triggered by changing this line to:
| disks: (0..10) | |
| disks: (0..1) |
Basically:
- Have a request to provision many disks
- Concurrently, eliminate one disk's viability
- Observe that walking the remaining search space shouldn't take a terribly long time.
(FWIW, I think this could be a companion test to this existing one, if you want to still test both behaviors: one where the concurrent request consumes all U.2s, and one where it only consumes one of them)
There was a problem hiding this comment.
we could probably even be proptesting across all possible local disk counts, although i don't know if that would really get us anything more useful than just having 0..1 and 0..10 test cases
There was a problem hiding this comment.
yeah I think the trickiness here is that we actually do handle both cases, but I think the performance profile is the thing that varies significantly between them
There was a problem hiding this comment.
Added this test in 4b2c631, it finished in about 10 seconds on my workstation
There was a problem hiding this comment.
hm, i wonder if we ought to be running these as benchmarks so that we can spot regressions in performance. not a blocker for this PR, since I think that may require some unrelated infrastructural work.
| match zpools_for_sled.get(&candidate_dataset.pool_id) { | ||
| Some(zpool_for_sled) => { | ||
| // Zpool still exists, does it still have room? | ||
|
|
||
| if !zpool_for_sled.has_room_for_allocation( | ||
| allocation.request.required_dataset_size(), | ||
| ) { | ||
| // Do not retain this as a candidate dataset, it no | ||
| // longer has room. | ||
| removed += 1; | ||
| return false; | ||
| } | ||
| } | ||
|
|
||
| None => { | ||
| // Zpool doesn't exist anymore! | ||
| removed += 1; | ||
| return false; | ||
| } | ||
| } |
There was a problem hiding this comment.
style nit, take it or leave it: we could maybe reduce "rightward drift" by one level of indentation here by writing:
| match zpools_for_sled.get(&candidate_dataset.pool_id) { | |
| Some(zpool_for_sled) => { | |
| // Zpool still exists, does it still have room? | |
| if !zpool_for_sled.has_room_for_allocation( | |
| allocation.request.required_dataset_size(), | |
| ) { | |
| // Do not retain this as a candidate dataset, it no | |
| // longer has room. | |
| removed += 1; | |
| return false; | |
| } | |
| } | |
| None => { | |
| // Zpool doesn't exist anymore! | |
| removed += 1; | |
| return false; | |
| } | |
| } | |
| let Some(zpool_for_sled) = zpools_for_sled.get(&candidate_dataset.pool_id) | |
| else { | |
| // Zpool doesn't exist anymore! | |
| removed += 1; | |
| return false; | |
| }; | |
| // Zpool still exists, does it still have room? | |
| if !zpool_for_sled.has_room_for_allocation( | |
| allocation.request.required_dataset_size(), | |
| ) { | |
| // Do not retain this as a candidate dataset, it no | |
| // longer has room. | |
| removed += 1; | |
| return false; | |
| } |
IMO that reads a little nicer because the logic flows top-to-bottom instead of kind of middle-out?
There was a problem hiding this comment.
personally I like the way it reads with the match :)
| affinity: None, | ||
| ncpus: 2, | ||
| memory: external::ByteCount::from_gibibytes_u32(16), | ||
| disks: (0..10) |
There was a problem hiding this comment.
we could probably even be proptesting across all possible local disk counts, although i don't know if that would really get us anything more useful than just having 0..1 and 0..10 test cases
hawkw
left a comment
There was a problem hiding this comment.
I think this looks pretty good! I'd want a +1 from Sean as well, though.
| .internal_context("selecting multiple disks failed") | ||
| })? | ||
| .into_iter() | ||
| .map(|disk| (disk.id(), disk)) |
There was a problem hiding this comment.
this makes me feel like the HashMap ought to be an iddqd::IdHashMap (or IdOrdMap), with an IdHashItem implementation for db::model::Disk, so that the invariant that the key is always the disk ID can be inherent in the data structure rather than implicit. not a huge deal since this hash map is relatively short lived and isn't mutable once it's collected, but it still feels a bit tidier.
| // Is it possible the disk has been hard-deleted somehow? | ||
| // Otherwise how would we land here, given that we just created | ||
| // the map! | ||
| return Err(Error::internal_error(&format!( | ||
| "disk id {disk_id} not found in map" | ||
| ))); |
There was a problem hiding this comment.
hm, i wonder whether we ought to be handling this by logging a warning and treating it as equivalent to the disk being deleted?
There was a problem hiding this comment.
right now, disks shouldn't ever be hard-deleted, so I think this should be a 500 level error.
There was a problem hiding this comment.
that's fair, i was just thinking about defensive programming in case someone or something is messing with the database manually.
| match zpools_for_sled.get(&candidate_dataset.pool_id) { | ||
| Some(zpool_for_sled) => { | ||
| // Zpool still exists, does it still have room? | ||
|
|
||
| if !zpool_for_sled.has_room_for_allocation( | ||
| allocation.request.required_dataset_size(), | ||
| ) { | ||
| // Do not retain this as a candidate dataset, it no | ||
| // longer has room. | ||
| removed += 1; | ||
| return false; | ||
| } | ||
| } | ||
|
|
||
| None => { | ||
| // Zpool doesn't exist anymore! | ||
| removed += 1; | ||
| return false; | ||
| } | ||
| } |
|
|
||
| impl ByteCount { | ||
| // None of these three constructors can create a value larger than | ||
| // `i64::MAX`. (Note that a `from_tebibytes_u32` could overflow u64.) |
There was a problem hiding this comment.
Your additions to this file inviolate this comment, and allow overflow?
Do we truly need this change? It seems unsafe
| CompleteLocalStorageAllocationLists::new( | ||
| &logctx.log, | ||
| config.sleds[0].sled_id, | ||
| config.instances[0].id, |
There was a problem hiding this comment.
I'm not sure I understand what's happening in this test.
instance_2_local_storage_diskslists disks fromconfig.instances[1].id- ... but this constructor uses
config.instances[0].id?
I think this means that the disk attachment check (our in-prod code) at the top of prune_invalidated_allocation_lists calls self.queue.clear() and doesn't actually check what this PR is trying to check.
I think local_storage_allocation_iterator_pruning_2 is doing the same thing below (gathering disks from instance 1, then creating a CompleteLocalStorageAllocationLists for instance 0).
Should these instances match?
There was a problem hiding this comment.
yes! this is an awful typo, fixed in 78381fd
| if removed > 0 { | ||
| info!(self.log, "removed {removed} from candidate datasets"); | ||
| } | ||
| } |
There was a problem hiding this comment.
So the check here covers this issue at construction-time, but my concern is about that infeasible state arising mid-search.
The new pruning logic added in this PR can change the candidate lists into a state that our constructor would have rejected, but it doesn't get a chance to.
Specifically:
- If
candidate_datasetsbecomes empty (which we check in the constructor here), or - The number of distinct pools across all candidate lists is now smaller than the number of requests (which we normally check in the constructor here)
If either of these happen as an outcome of prune_invalidated_allocation_lists, we never re-check them. I'm mostly worried about wasted work in https://github.com/jmpesp/omicron/blob/367f71137c72d64ab13ad8400a0cbe377f24eb27/nexus/db-queries/src/db/datastore/sled.rs#L775
NORMALLY I wouldn't care much; our checks should identify these cases and eliminate them anyway, but the next function in impl<'a> Iterator for CompleteLocalStorageAllocationLists<'a> is an extremely expensive operation: unless the allocation has already succeeded, it's generating quite a bit of additional queue work.
I made a separate comment below about the ..._pruning_{1,2} tests passing the wrong instance IDs - but I tried to check out this branch, fix that, and then compare the cost of "how expensive it it to call CompleteLocalStorageAllocationLists::next() in the "pruning_2" test with / without the check I suggested?
- The PR, as-is (with the correct instances patched):
next()takes 3.34 seconds of pure CPU time. - With the mid-search re-check I suggested:
next()takes 979 nanoseconds, because it's not spending time throwing things into / out of the queue.
A few weeks ago we were debugging an exhaustive search through all local storage allocations, and determined it was caused by the invalid PgNumeric conversion to ByteCount. The next step was to cause that search to bail out in order to unwind the saga: this was supposed to be achieved by adjusting
size_usedcolumn of the rendezvous local storage dataset to a number that would not be affected by the invalid conversion, and we did this, but observed that the search continued anyway!Consider the following incomplete allocation list:
Note the allocations and zpools here are ordered to make the example easier but will actually be random depending on the order of rows returned.
The remaining allocations to perform are 8 and 9, and they have a list of zpools to try to allocate against. The pathologically bad case is if any zpool could be a candidate:
Previous to this commit, the pruning step only affected the list of incomplete allocations. This doesn't cause the search to bail out! If zpool6 runs out of space due to concurrent allocations, the pruning step will not retain incomplete allocations that use zpool6:
but the iterator will continue to descend down the search space using zpool6 because it's still in the candidate list:
This commit adds a pruning step that removes from an allocation's candidate list based on the updated ZpoolGetForSledReservationResult information.